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display area located between a file management tool 
bar(116) and the handwritten input area (104). 
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AUTOMATICALLY SCROLLING HANDWRITTEN 
INPUT USER INTERFACE FOR PERSONAL DIGITAL 
ASSISTANTS AND THE LIKE 

5 DESCRIPTION 

RELATED APPLICATION 

The present invention is related to U.S. Patent Application No. 
09/901,878 (Attorney Docket No. LX00071) entitled "Handwriting Interface 
10 for Personal Digital Assistants and the like" to Seni et aL, assigned to the 
assignee of the present invention and filed July 9, 2001. 

BACKGROUND OF THE INVENTION 

15 Field of the Invention 

The present invention is related to personal digital assistants (PDAs) 
and more particularly to a user input interface for a PDA or the like. 



20 



Background Description 



Portable computing devices, such as what is normally referred to as a 
personal digital assistant (PDA), are increasing in popularity. A typical PDA is 
a limited function microcomputer provided with a pressure sensitive liquid 
crystal diode (LCD) display (a touch pad or a touch screen) for input and 
25 output (I/O). PDAs are being adapted for wireless Internet communication, 
e.g., using a modem for e-mail and web browsing. Further, for text input 
PDAs are known that have a specialized stroke based alphabet interface, e.g., 
Graffiti®, a selectable on-screen QWERTY keypad, or an expansion pack 
keyboard. 
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As these portable devices become smaller and more specialized, text 
input has become more difficult and less practical. Typical prior art 
handwriting recognition software may require users to learn special characters 
or effect a handwriting style in order to enter text. Text input using the 
5 Graffiti® unistroke (i.e., written with a single pen trace) alphabet can be un- 
natural because it requires users to adhere to strict rules that restrict character 
shapes; text input using an on-screen QWERTY keypad is somewhat clumsy 
because only small reductions in size can be made to keyboards before they 
become awkward to use. An expansion keyboard is impractical for on-the-go 

10 input. With either, the tapping on individual characters or the typing is less 
desirable than being able to handwrite notes or messages. Meanwhile, the 
demand for PDA information exchange, e-mail and internet access requires 
entry and retrieval of increasing amounts of data with the handheld device. 
Handwriting recognition (HWR) programs capable of dealing with 

15 natural (i.e., unrestricted in style) handwritten input are being developed to add 
to function and usefulness to PDAs and are crucial to the growth of mobile 
computing in the communications field. Handwriting recognition software, 
such as Transcriber (formerly known as CalliGrapher) from Microsoft Corp., 
allows the user to write anywhere on the screen, including on top of any 

20 displayed application and system elements. After a time-out period following 
a pen-lift, the digital ink is removed from the screen and then recognized, the 
recognition results are then displayed on the screen as ASCII text, and the next 
sentence or string of words can be handwritten on the screen. 

For small screen sized devices such as of PDAs, what are typically 

25 referred to as write-anywhere user interfaces, such as used by Transcriber, 
allow users to write two or three lines at a time at most, with at best two or 
three words each, which limits entry and prevents continuous uninterrupted 
longhand entry. Further, these write-anywhere interfaces are problematic 
because it is difficult to differentiate whether the stylus is acting as a pointer, 
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for clicking on application icons and the like, or an inking instrument for text 
entry. A common solution involves an un-natural "tap and hold" scheme 
wherein the pen has to be maintained down without dragging it for a certain 
amount of time in order to get the stylus to act temporarily as a mouse. This 
5 can lead to text input errors and the attendant aggravation and input delays 
caused by such errors. 

Another problem with a write-anywhere user interface is that fingers, as 
the writer is moving his/her hand through the screen, can often interfere with 
the (pressure-based) pen tracking mechanism. Simultaneous pressure from the 

10 stylus and a carelessly positioned pinky finger can cause the device to 

mislocate the intended stylus entry point, e.g., the device may use the average 
of the two contact locations. One solution to these problems has been 
provided by Seni et al., where a special area at the bottom of the screen is used 
for handwritten input, inputting one word at a time. 

15 Thus, there is a need for handwriting input user interface that allows 

continuous handwritten input, entering arbitrarily long streams of text on small 
palm-sized devices without interruption, i.e., without requiring the writer to 
stop after every character or word, and free from the ambiguities of write- 
anywhere interfaces. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 



The foregoing and other objects, aspects and advantages will be better 
understood from the following detailed preferred embodiment description with 
25 reference to the drawings, in which: 

Figures 1A&B are plan views of a handheld device in a preferred form 
of the invention showing a graphical handwriting user interface having a 
scrolling text input screen area; 
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Figure 2 is another example of a preferred embodiment handheld 
device including a pop-up list word recognition choices; 

Figure 3 shows an example of a state diagram of the preferred 
embodiment HIUI; 

5 Figure 4 is an example of a flow diagram for receiving continuous 

handwritten input according to the preferred embodiment of the present 
invention. 

DESCRIPTION OF PREFERRED EMBODIMENTS 
10 OF THE INVENTION 

The present invention is a method of interfacing with, and a 
handwritten input user interface (HIUI) for, small (shirt-pocket sized) portable 
devices with a touch-enabled input/output (I/O) screen, such as are commonly 

15 known as personal digital assistants (PDAs). The I/O screen includes a 
handwritten input I/O screen area that behaves as a "treadmill," receiving 
streams of handwritten input strokes and immediately displaying a duplicate of 
each stroke as digital ink therebeneath, while simultaneously moving digital 
ink strokes from right to left as the user handwrites text on what appears to be 

20 a virtual "ticker-tape." Thus, the user is presented with illusion of never 
running out of writing space. The streaming speed of the treadmill or ticker 
tape I/O automatically adapts to writing speed in a natural way such that the 
entry point remains, roughly, at the same horizontal location during each 
handwritten entry. Li this manner, unlike prior interfaces wherein space 

25 constraints limit the ability to continuously write on the device screen and thus 
slows text input, the present interface speeds throughput. A user is not 
required to wait for more screen space after the screen is filled because clear 
screen space is continuously made available by the scrolling action provided in 
the screen input area. 
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The portable devices may be capable of wireless message transmission 
(such as for web browsing and/or e-mail). The user interface of the present 
invention is typically in software and loaded into PDA storage. A state of the 
5 art handwriting recognition engine also is included in software. The 

handwriting user interface of the present invention enhances the usability, 
flexibility and power of the handheld device in which it is installed. An entire 
message may be quickly handwritten, converted to text as it is written, stored 
and then, transmitted, for example. 

10 Handwritten entries may be made continuously in a designated input 

area on the touch screen. The preferred location of the input area is the lower 
portion of the screen, so as to only partially block view of any application 
currently running on the device. Handwritten text is continuously entered onto 
a virtual ticker-tape in the input area, using a stylus and requiring minimal 

15 horizontal stylus deflection during entry. If recognition is included, the 

recognition engine demarcates word boundaries to identify individual words 
within the input stream as well as provide word recognition. Recognition 
results can be displayed in the normal display area of the screen above the 
input area and the input area scrolls automatically, behaving as treadmill, or 

20 ticker tape moving from right to left automatically adapting to Writing speed. 
Continuous or continuously is used herein with respect to handwritten input to 
mean that pauses are not required between words, i.e., without requiring the 
user to stop after each word during entry. Of course, the user may stop or 
pause as the user desires, but this is irrespective of and in no way restrictive of 

25 the present invention. Thus, a user may continuously write (enter) a phrase, a 
sentence, a paragraph, a page, a book (as long as the input device has enough 
memory) or any combination thereof, free from pauses that might otherwise be 
artificially imposed by system constraints. 
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Figures 1 A-B are an example of a preferred embodiment handheld 
device 100 with a graphical handwriting user interface 102 according to 
preferred embodiment of the present invention. A lower portion of the display 
is the designated handwriting input area 104 with the area 105 thereabove used 
5 for displaying the recognized text that is input in area 104, as well as for other 
features of the various applications the device incorporates. Action icons 106, 
108, 110, 112 and 114 are disposed at a right side of the handwriting user 
interface 102. Recognized text is displayed in the screen area 105 between a 
file management tool bar 116 and the handwriting input area 104. In this 
10 embodiment, a scroll bar 1 18 is disposed at the right side of the interface 
display 1 12. As text is entered, the entry is displayed contemporaneously as 
digital ink in the input area 104 as can be seen in figure 1 A. Effectively the 
user is provided with the sense that the writing area is moving from right to 
left. 

15 Optionally, a vertical word separator line 120 (a word demarcation for 

signaling the end of one word and the beginning of a next word) may be 
selectively displayed. Provided input continues to the left of the word 
separator line 120, the current word is continued. Otherwise, if input continues 
to the right of the word separator line 120, a word break is indicated and a new 

20 word is begun. If the word separator line 120 is not used or included, word 
demarcation may be done, simply, by one or more spaces or other special 
characters, intentionally or implicitly inserted between a current word and a 
next word. 

As the user writes, the digital ink trails off to the left on the virtual 
25 ticker tape, as can be seen in Figure IB. Accordingly, when the user has 

finished writing the characters such as those in the illustrated word "happy," it 
will be shifted so that it gradually is removed from the input area 104 as by 
trailing off, e.g., off from the right to the left side. Also, the word separator 
line 120, if included and used, likewise trails off to the left edge of the input 
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area 104 and remains there until entry resumes for the next word, "words" in 
this example. Otherwise, if the word separator line 120 is not used/included, 
the current word may still trail off with a next word begun by allowing a 
sufficient gap between the current word and the new entry such that the gap is 
5 recognized as a space. Alternately, a special character or a special space 
character may be included to signify a word break. By virtue of each word 
trailing off and being removed from the input area, the user is presented with 
substantially unlimited writing space in the normally space-constrained input 
area 104. 

10 The device 100 may include a communications function and, to that 

end in this embodiment, an antenna 122 is shown at the top of the device 100. 
Individual function switches, buttons and other controls are disposed about the 
device, as is deemed appropriate for the particular device. The device 100 may 
also include an expansion port 124 or an expansion port function may be 

15 provided wirelessly through antenna 122. Preferably, the device 100 runs 
under a state of the art operating system for such handheld devices, e.g. 
Windows® CE from Microsoft Corporation, Epoc® from Symbian or the Palm 
OS® from Palm, Inc. 

The preferred embodiment HTUI of the present invention may be 

20 employed with a handwriting recognition engine capable of recognizing 

handwritten text, continuously input using any combination of writing styles. 
The user is not restricted to cursive entry, nor is each entry delineated by a 
pause or by a time out between entries. Handwritten input may be provided in 
cursive (i.e., contiguous characters in each entry touching or connected), pure 

25 print (i.e., characters in every entry disconnected and not touching), 

pseudo-print (at most, pairs of characters in entries touch) or any combination 
thereof. Preferably, the recognition engine is the QuickPrintPro™ engine from 
Motorola, Inc., Lexicus Division. The recognition engine includes a main 
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dictionary and may also include a user dictionary to which the user may add 
words to supplement the main dictionary. 

In sentence or continuous mode, the recognition engine automatically 
separates or brackets each word as it is entered, using spaces, special characters 
5 or word separator line 120 to bracket individual separate words. The 

recognition engine takes each individual word on the fly, and compares that 
handwritten input word against all words contained in the main dictionary and 
the user dictionary. A probability score is generated by the recognition engine 
for each dictionary word which is indicative of the likelihood that the 

10 handwritten entry matches that particular dictionary word. Based on each 

word's probability score, a list of likely matches is collected. When a pen trace 
falls out of the visible window area, the recognizer is invoked to buffer or 
recognize the ink. That is, the recognizer may, or may not, return a result at 
that time. At the very end of writing (say after a time-out), the recognition 

15 engine is requested to process any buffered ink for which no recognition result 
has been returned. 

Figure 2 is another example of a preferred embodiment handheld 
device 126 including a pop-up list 128 of recognition choices. The pop-up list 
128 is available on "demand" — that is, if the user selects a word in the text 

20 buffer (such as a word corresponding to one of m previous recognition results 
for which a list of possible secondary results still remains). The pop-up list 
128 is available only when both the application receiving the input is running 
in the foreground and the recognition engine is tightly coupled to it. From the 
list of recognition results, the handwriting recognition engine calculates a 

25 confidence level for the one word (the primary word) with the highest 
probability. If that confidence level exceeds a preselected or confidence 
threshold, it is taken as an indication that the word with the highest probability 
is in fact correct and the highest scoring word is displayed as the primary word 
choice. All other results are referred to as secondary word choices and may be 
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included in the pop-up list 128. Every recognition result is provided with a list 
of a primary word, if any, and secondary word choices that are made available 
for the receiving application to show upon user request, e.g., by selecting a 
displayed recognition result. So, if the confidence level is above the 
5 preselected threshold, the HUI automatically loads a primary word choice into 
the device* s input buffer for delivery to the active application. Otherwise, 
when the confidence level of the primary word choice is below the confidence 
threshold, an indication is provided that the recognition engine cannot find a 
likely candidate, e.g., by loading "???" or something similar into the device's 

10 input buffer for delivery to the active application. 

Figure 3 is an example of a state diagram 130 of the preferred 
embodiment user interface which includes several individual objects or 
functions and may be implemented in any suitable programming language, 
such as tool command language (tcl) for example. A main ink area object is 

15 included, state 132, to handle ink collection and display collected results in the 
input window from other independent functions. When the stylus is depressed, 
moving to state 134, inking begins. As long as the stylus moves on the I/O 
screen 104 and the stylus is down, state 136, ink is added to the screen and the 
I/O screen 104 scrolls automatically in state 138. The scrolling speed is 

20 checked and set in state 140. If the stylus is raised, state 142, a pause is 

initiated at wait state 144. If a pause is long enough to indicate that an entry 
has been completed, then, entering text recognition state 146, the previous 
completed entry is recognized as text and displayed in the display area 105. In 
addition, whenever a pen trace falls out of the visible window area, again the 

25 recognizer is invoked. The recognizer may return a result at that time and 
return to state 132 to await the next entry. Otherwise, the recognizer may 
recognize that the current entry is incomplete and return to scroll state 138. 
That is, in response to a recognition call, the HUI may simply buffer the ink 
and wait for additional ink to arrive. If a pause during entry is short enough 
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such that end of input is not indicated, e.g., the user is waiting for the ink to 
scroll off the screen to print the next letter; then, the VO display area 104 auto 
scrolls in state 138 and, the scroll speed is updated, state 140. 

Thus, an ink area object is included which creates a virtual canvas in 
5 the PDA screen for digital ink input, display and text recognition. The ink area 
object includes numerous functional objects, each providing a function for 
inking input/display/recognition. An initialization functional object initializes 
the inking area and inking area variables. A timer functional object provides a 
timing function for timing delays between beginning strokes and ending text 

10 entries for example. A scrolling object sets the scroll rate for the ticker tape 
type ink display. A vertical marking object locates a stylus marker within the 
inking area. Ink width can be varied with a set ink width object. A deletion 
object is included to provide, for example, a back space function. Ink strokes 
are added to the display by an add ink point object. Touching the stylus to the 

15 screen in the handwritten input area is treated analogous to pressing a mouse 
button and moving the mouse, dragging an object across a computer screen 
with the object path leaving a trail of digital ink. So, button-type control and 
commands are provided by button function objects such as, a button down, 
button up or a button motion function. In addition, a button timeout object 

20 determines if a button function response was missed or is delayed beyond a 

reasonable amount of time. A clear ink object and a word delete object allow a 
user to restart inking on an entry, for example. A recognition object may be 
included, as described above, to alphanumerically convert handwritten entries 
in the input area to text and display the text. Finally, a scrolling animation 

25 object controls auto-scrolling in inking the area, i.e., scrolling start and stop as 
well as initialization. Also, the scrolling animation object determines whether 
or not pairs of individual input entries are separated by a space. 
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Figure 4 is an example of a flow diagram 150 for receiving continuous 
handwritten input according to the preferred embodiment of the present 
invention. In step 152 the user provides handwritten input, writing in the 
handwritten input area and in step 154, as the user enters that handwritten 
5 input, digital ink is displayed in the handwritten input area. In step 156 the 
user stroke rate is determined and the step 158 the scroll rate is matched to the 
stroke rate. In step 160 the end of word separator, e.g., end of word separator 
line 120, is checked and if an end of word is found, then if word recognition is 
included, word recognition begins in step 162. Alternatively, an end of word 

10 may be recognized by the end of a current entry scrolling past the edge of the 
input field, giving the illusion of the "ink falling off of the screen" followed by 
sufficient passage of time, i.e., by insertion of a blank character after the word. 
In step 164 the recognition results from word recognition are displayed in the 
display area of the screen. Regardless of whether an end of word is found in 

15 step 160, handwritten input may continue in step 152. 

Thus, the I/O screen of the preferred embodiment provides a 
handwritten input I/O screen area that behaves as a "treadmill," capable of 
receiving continuous handwritten input strokes and immediately displaying a 
duplicate of each stroke as digital ink therebeneath, automatically scrolling 

20 from right to left as the user handwrites entries on what appears to be a virtual 
"ticker-tape." The user is presented with illusion of endless writing space. 
The streaming speed of the I/O treadmill or ticker tape automatically adapts to 
user's writing speed in a natural way. 

While the invention has been described in terms of preferred 

25 embodiments, those skilled in the art will recognize that the invention can be 
practiced with modification within the spirit and scope of the appended claims. 
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CLAIMS 

We claim: 

5 1 . An electronic device for handwritten input and 

contemporaneous display of said handwritten input, the device comprising: 

a housing having a touch input screen thereon for handwritten input by 
engagement therewith; 

a recognition engine that recognizes the handwritten input and converts 
10 the recognized input to text for display on the screen; and 

scrolling software that causes at least a portion of the screen to appear 
to move as handwritten entries are input thereon so as to continuously present 
screen space on the screen to the user for input thereby maximizing character 
input rates. 

15 

2. The electronic device of claim 1 wherein the screen is of a 
predetermined size and the screen portion is smaller than the predetermined 
screen size so that only the screen portion appears to move during handwritten 
input. 

20 

3. The electronic device of claim 1 wherein the screen has an input 
area including the screen portion on which handwritten text is input and 
corresponding digital ink is displayed, and an output area separate from the 
input area on which recognized text is displayed. 

25 

4. The electronic device of claim 3 wherein the housing includes 
electronic circuitry coupled to the screen with the recognition engine operable 
by engagement with the screen in the input area and inoperable by engagement 
with the screen in the output area. 
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5. The electronic device of claim 3 wherein the housing includes 
5 electronic circuitry coupled to the screen, and 

an input device that when engaged with the screen input area 
cooperates with the circuitry to act as a digital pen for inputting text and when 
engaged with the screen output area cooperates with the circuitry to act as a 
pointer for clicking 
10 functions. 

6. A computer program product for inputting handwritten entries 
into a computer, said computer program product comprising a computer usable 
medium having computer readable program code thereon, said computer 

15 readable program code comprising: 

computer readable program code means for continuously receiving 
handwritten entries; 

computer readable program code means for converting said 
handwritten entry into digital ink; 
20 computer readable program code means for setting a scrolling speed 

responsive to an entry input rate; and 

computer readable program code means for displaying newly entered 
said digital ink and removing previously displayed digital ink from a display at 
a rate set by said controlling speed, displayed said digital ink appearing as if on 
25 a ticker tape. 



7. A computer program product for inputting handwritten entries 
into a computer as in claim 6 further comprising computer readable program 
code means for displaying a word separator. 
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8. A computer program product for inputting handwritten entries 
into a computer as in claim 6 further comprising: 

computer readable program code means for identifying individual 
words and calling handwriting recognition; and 
5 computer readable program code means for recognizing handwritten 

words and providing recognized said words to a display. 



9. A computer program product for inputting handwritten entries 
into a computer as in clam 8 wherein the computer readable program code 
10 means for receiving handwritten entries further comprises: 

computer readable program code means for determining whether said 
handwritten entries are being made in an input area of a touch sensitive screen 
or in an other area of said touch sensitive screen than said input area; and 
computer readable program code means for providing commands 
15 responsive to entries in said other area, entries in said input area being received 
as handwritten entries. 
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